Ch.12 Orthogonal Projection

Return to TOC

Onto a Line


Imagine "walking" from the tip of v⃗\vec{v} to the line in an orthogonal fashion. Since the line is the span of some vector l={c⋅s⃗ ∣ c∈R}l=\{c\cdot\vec{s}\space|\space c\in\mathbb{R}\}, we're looking for some cpc_p so that cps⃗c_p\vec{s} is orthogonal to v⃗−cps⃗\vec{v}-c_p\vec{s}

To solve, notice that v⃗−cps⃗\vec{v}-c_p\vec{s} is orthogonal to s⃗\vec{s} itself, so
s⃗⋅(v⃗−cps⃗)=0→s⃗⋅v⃗−cps⃗2=0→cp=s⃗⋅v⃗s⃗⋅s⃗\vec{s}\cdot(\vec{v}-c_p\vec{s})=0\rightarrow \vec{s}\cdot\vec{v}-c_p\vec{s}^2=0\rightarrow c_p=\frac{\vec{s}\cdot\vec{v}}{\vec{s}\cdot\vec{s}}
Thus, we have the orhtogonal projection of v⃗\vec{v} onto l=[s⃗]l=[\vec{s}] is
proj[s⃗](v⃗)=v⃗⋅s⃗s⃗⋅s⃗⋅s⃗\text{proj}_{[\vec{s}]}(\vec{v})=\frac{\vec{v}\cdot\vec{s}}{\vec{s}\cdot\vec{s}}\cdot\vec{s}
This vector w⃗=proj[s⃗](v⃗)\vec{w}=\text{proj}_{[\vec{s}]}(\vec{v}) is the only vector in the line [s⃗][\vec{s}] such that v⃗−w⃗\vec{v}-\vec{w} is orthogonal to any vector in [s⃗][\vec{s}]

Example 12.1

The projection of this R3\mathbb{R}^3 vector into the line
v⃗=(311)  L={c⋅(1−21) ∣ c∈R}\vec{v}=\begin{pmatrix}3\\1\\1\end{pmatrix}\space\space L=\{c\cdot\begin{pmatrix}1\\-2\\1\end{pmatrix}\space|\space c\in\mathbb{R}\}
is the vector
projLv⃗=(311)⋅(1−21)(1−21)⋅(1−21)⋅(1−21)=26(1−21)=(1/3−2/31/3)\text{proj}_L{\vec{v}}=\frac{\begin{pmatrix}3\\1\\1\end{pmatrix}\cdot\begin{pmatrix}1\\-2\\1\end{pmatrix}}{\begin{pmatrix}1\\-2\\1\end{pmatrix}\cdot\begin{pmatrix}1\\-2\\1\end{pmatrix}}\cdot\begin{pmatrix}1\\-2\\1\end{pmatrix}=\frac{2}{6}\begin{pmatrix}1\\-2\\1\end{pmatrix}=\begin{pmatrix}1/3\\-2/3\\1/3\end{pmatrix}

Gram-Schmidt Orthogonalization

Notice how v⃗\vec{v} can be decomposed into
v⃗=proj[s⃗](v⃗)+(v⃗−proj[s⃗](v⃗))\vec{v}=\text{proj}_{[\vec{s}]}(\vec{v})+\left(\vec{v}-\text{proj}_{[\vec{s}]}(\vec{v})\right)
These are orthogonal, and can be seen as "non-interacting," i.e. linearly independent

Vectors v⃗1,...,v⃗k∈R\vec{v}_1,...,\vec{v}_k\in\mathbb{R} are mutually orthogonal if any pair of them are orthogonal
i.e. for any i≠ji\ne jv⃗i\vec{v}_i and v⃗j\vec{v}_j are orthogonal
For example, the standard basis vectors are mutually orthogonal

If the vectors in a set {v⃗1,...,v⃗k}⊂Rn\{\vec{v}_1,...,\vec{v}_k\}\subset\mathbb{R}^n are mutually orthogonal and nonzero, then the set is linearly independent.

Proof

Consider c1v⃗1+⋯+ckv⃗k=0c_1\vec{v}_1+\cdots+c_k\vec{v}_k=0. For i∈{1,...,k}i\in\{1,...,k\}, taking the dot product on both sides gives
v⃗i⋅(c1v⃗1+⋯+ckv⃗k)=v⃗i⋅0⟹ci(v⃗i⋅v⃗i)=0\vec{v}_i\cdot(c_1\vec{v}_1+\cdots+c_k\vec{v}_k)=\vec{v}_i\cdot0\\ \implies c_i(\vec{v}_i\cdot\vec{v}_i)=0
Since v⃗i≠0\vec{v}_i\ne0, we must have v⃗i⋅v⃗i≠0\vec{v}_i\cdot\vec{v}_i\ne0, therefore ci=0c_i=0.
Since all ci=0c_i=0, the set is linearly independent.

A corollary of this is that kk mutually orthogonal vectors of a kk-dimensional vector space is a basis, because a subset of kk linearly independent vectors of a kk-dimensional space is a basis.
An orthogonal basis for a vector space is a basis of mutually orthogonal vectors

Gram-Schmidt Orthogonalization

If ⟨β⃗1,...,β⃗k⟩\langle\vec{\beta}_1,...,\vec{\beta}_k\rangle is a basis for a subspace of Rn\mathbb{R}^n, then the vectors
κ⃗1=β⃗1κ⃗2=β⃗2−proj[κ⃗1](β⃗2)κ⃗3=β⃗3−proj[κ⃗1](β⃗3)−proj[κ⃗2](β⃗3)⋮κ⃗k=β⃗k−proj[κ⃗1](β⃗k)−⋯−β⃗3−proj[κ⃗k−1](β⃗k)\begin{array}{rcl} \vec{\kappa}_1&=&\vec{\beta}_1\\ \vec{\kappa}_2&=&\vec{\beta}_2-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_2)\\ \vec{\kappa}_3&=&\vec{\beta}_3-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_3)-\text{proj}_{[\vec{\kappa}_2]}(\vec{\beta}_3)\\ &\vdots\\ \vec{\kappa}_k&=&\vec{\beta}_k-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_k)-\cdots-\vec{\beta}_3-\text{proj}_{[\vec{\kappa}_{k-1}]}(\vec{\beta}_k) \end{array}
form an orthogonal basis for the same subspace. Moreover,
span(κ⃗1,...,κ⃗i)=span(β⃗1,...,β⃗i)\text{span}({\vec{\kappa}_1,...,\vec{\kappa}_i})=\text{span}({\vec{\beta}_1,...,\vec{\beta}_i}) for all i=1,...ki=1,...k

Proof

We use induction to show that each κi\kappa_i:

Case i=1i=1: this is trivial
Case i=2i=2: we have
κ⃗2=β⃗2−proj[κ⃗1](β⃗2)=β⃗2−β⃗2⋅κ⃗1κ⃗1⋅κ⃗1⋅κ⃗1=β⃗2−β⃗2⋅κ⃗1κ⃗1⋅κ⃗1⋅β⃗1\vec{\kappa}_2=\vec{\beta}_2-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_2)=\vec{\beta}_2-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{\vec{\kappa}_1\cdot\vec{\kappa}_1}\cdot\vec{\kappa}_1=\vec{\beta}_2-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{\vec{\kappa}_1\cdot\vec{\kappa}_1}\cdot\vec{\beta}_1
This is nonzero because the β⃗\vec{\beta}'s are linearly independent, is clearly in the span ⟨β⃗1,β⃗2⟩\langle\vec{\beta}_1,\vec{\beta}_2\rangle, and is orthogonal to κ⃗1\vec{\kappa}_1 because the projection is orthogonal
Case i=3i=3: we have
κ⃗3=β⃗3−proj[κ⃗1](β⃗3)−proj[κ⃗2](β⃗3)=β⃗3−β⃗3⋅κ⃗1κ⃗1⋅κ⃗1⋅κ⃗1−β⃗3⋅κ⃗2κ⃗2⋅κ⃗2⋅κ⃗2=β⃗3−β⃗2⋅κ⃗1κ⃗1⋅κ⃗1⋅β⃗1−β⃗3⋅κ⃗2κ⃗2⋅κ⃗2⋅(β⃗2−β⃗2⋅κ⃗1κ⃗1⋅κ⃗1⋅β⃗1)\vec{\kappa}_3=\vec{\beta}_3-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_3)-\text{proj}_{[\vec{\kappa}_2]}(\vec{\beta}_3)\\=\vec{\beta}_3-\frac{\vec{\beta}_3\cdot\vec{\kappa}_1}{\vec{\kappa}_1\cdot\vec{\kappa}_1}\cdot\vec{\kappa}_1-\frac{\vec{\beta}_3\cdot\vec{\kappa}_2}{\vec{\kappa}_2\cdot\vec{\kappa}_2}\cdot\vec{\kappa}_2\\=\vec{\beta}_3-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{\vec{\kappa}_1\cdot\vec{\kappa}_1}\cdot\vec{\beta}_1-\frac{\vec{\beta}_3\cdot\vec{\kappa}_2}{\vec{\kappa}_2\cdot\vec{\kappa}_2}\cdot(\vec{\beta}_2-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{\vec{\kappa}_1\cdot\vec{\kappa}_1}\cdot\vec{\beta}_1)
This is nonzero and in the span ⟨β⃗1,β⃗2,β⃗3⟩\langle\vec{\beta}_1,\vec{\beta}_2,\vec{\beta}_3\rangle becuase they are linearly independent, and it is not hard to check that this is orthogonal to κ⃗1\vec{\kappa}_1 and κ⃗2\vec{\kappa}_2
Continue in this fashion to prove for all i=1,...,ki=1,...,k

Note that if ⟨β⃗1,...,β⃗k⟩\langle\vec{\beta}_1,...,\vec{\beta}_k\rangle is already orthogonal, the process just gives κ⃗i=β⃗i\vec{\kappa}_i=\vec{\beta}_i for i=1,...,ki=1,...,k

Example 12.2

Derive an orthogonal basis K=⟨κ⃗1,κ⃗2⟩K=\langle\vec{\kappa}_1,\vec{\kappa}_2\rangle for the basis
B=⟨(12),(13)⟩B=\langle\begin{pmatrix}1\\2\end{pmatrix},\begin{pmatrix}1\\3\end{pmatrix}\rangle

First, κ⃗1=β⃗1=(12)\vec{\kappa}_1=\vec{\beta}_1=\begin{pmatrix}1\\2\end{pmatrix}
Then,
κ⃗2=β⃗2−proj[κ⃗1](β⃗2)=(13)−(13)⋅(12)(12)⋅(12)⋅(12)=(−2/51/5)\vec{\kappa}_2=\vec{\beta}_2-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_2)=\begin{pmatrix}1\\3\end{pmatrix}-\frac{\begin{pmatrix}1\\3\end{pmatrix}\cdot\begin{pmatrix}1\\2\end{pmatrix}}{\begin{pmatrix}1\\2\end{pmatrix}\cdot\begin{pmatrix}1\\2\end{pmatrix}}\cdot\begin{pmatrix}1\\2\end{pmatrix}=\begin{pmatrix}-2/5\\1/5\end{pmatrix}
Thus, K=⟨(12),(−2/51/5)⟩K=\langle\begin{pmatrix}1\\2\end{pmatrix},\begin{pmatrix}-2/5\\1/5\end{pmatrix}\rangle
Note that because (12)⋅(−2/51/5)=0\begin{pmatrix}1\\2\end{pmatrix}\cdot\begin{pmatrix}-2/5\\1/5\end{pmatrix}=0, they are orthogonal

Example 12.3

Derive an orthogonal basis KK for
B=⟨(112),(−121),(03−1)⟩B=\langle\begin{pmatrix}1\\1\\2\end{pmatrix},\begin{pmatrix}-1\\2\\1\end{pmatrix},\begin{pmatrix}0\\3\\-1\end{pmatrix}\rangle

κ⃗1=β⃗1=(112) \vec{\kappa}_1=\vec{\beta}_1=\begin{pmatrix}1\\1\\2\end{pmatrix}
κ⃗2=β⃗2−proj[κ⃗1](β⃗2)=(−121)−(−121)⋅(112)(112)⋅(112)⋅(112)=(−121)−12(112)=(−3/23/20) \vec{\kappa}_2=\vec{\beta}_2-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_2)=\begin{pmatrix}-1\\2\\1\end{pmatrix}-\frac{\begin{pmatrix}-1\\2\\1\end{pmatrix}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}}{\begin{pmatrix}1\\1\\2\end{pmatrix}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}=\begin{pmatrix}-1\\2\\1\end{pmatrix}-\frac{1}{2}\begin{pmatrix}1\\1\\2\end{pmatrix}=\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}
κ⃗3=β⃗3−proj[κ⃗1](β⃗3)−proj[κ⃗2](β⃗3)=(03−1)−(03−1)⋅(112)(112)⋅(112)⋅(112)−(03−1)⋅(−3/23/20)(−3/23/20)⋅(−3/23/20)⋅(−3/23/20) =(03−1)−16(112)−9/29/2(−3/23/20)=(4/34/3−4/3) \vec{\kappa}_3=\vec{\beta}_3-\text{proj}_{[\vec{\kappa}_1]}(\vec{\beta}_3)-\text{proj}_{[\vec{\kappa}_2]}(\vec{\beta}_3)= \begin{pmatrix}0\\3\\-1\end{pmatrix}-\frac{\begin{pmatrix}0\\3\\-1\end{pmatrix}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}}{\begin{pmatrix}1\\1\\2\end{pmatrix}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}}\cdot\begin{pmatrix}1\\1\\2\end{pmatrix}-\frac{\begin{pmatrix}0\\3\\-1\end{pmatrix}\cdot\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}}{\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}\cdot\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}}\cdot\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}\\\space\\ =\begin{pmatrix}0\\3\\-1\end{pmatrix}-\frac{1}{6}\begin{pmatrix}1\\1\\2\end{pmatrix}-\frac{9/2}{9/2}\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix}=\begin{pmatrix}4/3\\4/3\\-4/3\end{pmatrix}
So in summary,
K=⟨(112),(−3/23/20),(4/34/3−4/3)⟩K=\langle\begin{pmatrix}1\\1\\2\end{pmatrix},\begin{pmatrix}-3/2\\3/2\\0\end{pmatrix},\begin{pmatrix}4/3\\4/3\\-4/3\end{pmatrix}\rangle

The orthogonal basis KK can be normalized to have length 11, making it an orthonormal basis

A family of vectors in Rn\mathbb{R}^n is orthonormal if they are mutually orthogonal and all have length 1.
In other words, for i∈{1,...k}i\in\{1,...k\} with i<ji<j, {β⃗1,...,β⃗l}⊆Rn\{\vec{\beta}_1,...,\vec{\beta}_l\}\subseteq\mathbb{R}^n is orthonormal if β⃗i⋅β⃗j=0\vec{\beta}_i\cdot\vec{\beta}_j=0 and β⃗i⋅β⃗i=1\vec{\beta}_i\cdot\vec{\beta}_i=1
If it is also a basis, then it is an orthonormal basis

Summary of Gram-Schmidt process

Proof Since BMB_M is a basis for MM, we can write v⃗=c1b⃗1+⋯+ckb⃗k\vec{v}=c_1\vec{b}_1+\cdots+c_k\vec{b}_k with c1,...,ck∈Rc_1,...,c_k\in\mathbb{R}. To find cic_i take the dot product with b⃗i\vec{b}_i so v⃗⋅b⃗i=(c1b⃗1+⋯+cib⃗i+⋯+ckb⃗k)⋅b⃗i=c1b⃗1⋅b⃗1+⋯+cib⃗i⋯b⃗i+⋯+ckb⃗k⋅b⃗k=ci\begin{array}{lcl}\vec{v}\cdot\vec{b}_i&=&(c_1\vec{b}_1+\cdots+c_i\vec{b}_i+\cdots+c_k\vec{b}_k)\cdot\vec{b}_i\\&=&c_1\vec{b}_1\cdot\vec{b}_1+\cdots+c_i\vec{b}_i\cdots\vec{b}_i+\cdots+c_k\vec{b}_k\cdot\vec{b}_k\\&=&c_i\end{array} since b⃗i⋅b⃗j=0\vec{b}_i\cdot\vec{b}_j=0 for i≠ji\ne j and b⃗i⋅b⃗i=1\vec{b}_i\cdot\vec{b}_i=1

We will say w⃗∈Rn\vec{w}\in\mathbb{R}^n is orthogonal to subspace MM of Rn\mathbb{R}^n if it is orthogonal to every vector v⃗∈M\vec{v}\in M, i.e. w⃗⋅v⃗=0\vec{w}\cdot\vec{v}=0 for all v⃗∈M\vec{v}\in M
a) The only vector v⃗∈M\vec{v}\in M that is orthogonal to MM is 0⃗\vec{0}
b) If w⃗1\vec{w}_1 and w⃗2\vec{w}_2 are orthogonal to MM, then any c1w⃗1+c2w⃗2c_1\vec{w}_1+c_2\vec{w}_2 with c1,c2∈Rc_1,c_2\in\mathbb{R} is also orthogonal to MM
c) If BM=⟨β⃗1,...,β⃗k⟩B_M=\langle\vec{\beta}_1,...,\vec{\beta}_k\rangle is a basis for MM, then w⃗\vec{w} is orthogonal to MM iff w⃗⋅β⃗i=0\vec{w}\cdot\vec{\beta}_i=0 for all i=1,...,ki=1,...,k

Proofs

a) We must have v⃗\vec{v} orthogonal to itself, so
v⃗⋅v⃗=∣v⃗∣2=0⟹v⃗=0\vec{v}\cdot\vec{v}=|\vec{v}|^2=0\implies \vec{v}=0
b) We have w⃗1⋅v⃗=0\vec{w}_1\cdot\vec{v}=0 and w⃗2⋅v⃗=0\vec{w}_2\cdot\vec{v}=0 for all v⃗∈M\vec{v}\in M, so
(c1w⃗1+c2w⃗2)⋅v⃗=c1w⃗1⋅v⃗+c2w⃗2⋅v⃗=0(c_1\vec{w}_1+c_2\vec{w}_2)\cdot\vec{v}=c_1\vec{w}_1\cdot\vec{v}+c_2\vec{w}_2\cdot\vec{v}=0
c) If w⃗∈Rn\vec{w}\in\mathbb{R}^n is orthogonal to MM, then it is orthogonal to every b⃗i∈M\vec{b}_i\in M. Conversely, assume w⃗∈Rn\vec{w}\in\mathbb{R}^n is such that w⃗⋅b⃗i=0\vec{w}\cdot\vec{b}_i=0 for all i=1,...,ki=1,...,k.
Any vector v⃗∈M\vec{v}\in M can be represented as v⃗=c1b⃗1+⋯+ckb⃗k\vec{v}=c_1\vec{b}_1+\cdots+c_k\vec{b}_k, so
w⃗⋅v⃗=w⃗⋅(c1b⃗1+⋯+ckb⃗k)=c1w⃗⋅b⃗1+⋯+ckw⃗⋅b⃗k=0\vec{w}\cdot\vec{v}=\vec{w}\cdot(c_1\vec{b}_1+\cdots+c_k\vec{b}_k)=c_1\vec{w}\cdot\vec{b}_1+\cdots+c_k\vec{w}\cdot\vec{b}_k=0

Onto a Subspace

This is a generalization of the projection onto a line.

Let MM be a subspace of Rn\mathbb{R}^n, then for every vector w⃗∈Rn\vec{w}\in\mathbb{R}^n, there exists a unique vector v⃗∈M\vec{v}\in M such that w⃗−v⃗\vec{w}-\vec{v} is orthogonal to MM.
We denote v⃗=projM(w⃗)\vec{v}=\text{proj}_M(\vec{w}) and call it the orthogonal projection of w⃗\vec{w} on MM.
If BM=⟨b⃗1,...,b⃗k⟩B_M=\langle\vec{b}_1,...,\vec{b}_k\rangle is an orthogonal basis for MM, then
projM(w⃗)=(w⃗⋅b⃗1)b⃗1+⋯+(w⃗⋅b⃗k)b⃗k\text{proj}_M(\vec{w})=(\vec{w}\cdot\vec{b}_1)\vec{b}_1+\cdots+(\vec{w}\cdot\vec{b}_k)\vec{b}_k

Proof

The vector v⃗=(w⃗⋅b⃗1)b⃗1+⋯+(w⃗⋅b⃗k)b⃗k\vec{v}=(\vec{w}\cdot\vec{b}_1)\vec{b}_1+\cdots+(\vec{w}\cdot\vec{b}_k)\vec{b}_k is such that w⃗−v⃗\vec{w}-\vec{v} is orthogonal to MM. Since v⃗∈M\vec{v}\in M and BMB_M is an orthogonal basis, v⃗=(v⃗⋅b⃗1)b⃗1+⋯+(v⃗⋅b⃗k)b⃗k\vec{v}=(\vec{v}\cdot\vec{b}_1)\vec{b}_1+\cdots+(\vec{v}\cdot\vec{b}_k)\vec{b}_k.
Therefore,
v⃗⋅b⃗1=w⃗⋅b⃗1, ..., v⃗⋅b⃗k=w⃗⋅b⃗k\vec{v}\cdot\vec{b}_1=\vec{w}\cdot\vec{b}_1,\space...,\space\vec{v}\cdot\vec{b}_k=\vec{w}\cdot\vec{b}_k
This implies (w⃗−v⃗)b⃗i=0(\vec{w}-\vec{v})\vec{b}_i=0 for all i=1,...,ki=1,...,k, so by c) from before w⃗−v⃗\vec{w}-\vec{v} is orthogonal to MM.
Now suppose v⃗1,v⃗2∈M\vec{v}_1,\vec{v}_2\in M are such that w⃗−v⃗1\vec{w}-\vec{v}_1 and w⃗−v⃗2\vec{w}-\vec{v}_2 are orthogonal to MM. By b) from before, (w⃗−v⃗1)−(w⃗−v⃗2)=v⃗2−v⃗1(\vec{w}-\vec{v}_1)-(\vec{w}-\vec{v}_2)=\vec{v}_2-\vec{v}_1 is orthogonal to MM, but v⃗2−v⃗1∈M\vec{v}_2-\vec{v}_1\in M, so by a) v⃗2−v⃗1=0⟹v⃗2=v⃗1\vec{v}_2-\vec{v}_1=0\implies\vec{v}_2=\vec{v}_1
proving its uniqueness

Let MM be a subspace of Rn\mathbb{R}^n. The map projM:Rn→M,w⃗↦projM(w⃗)\text{proj}_M:\mathbb{R}^n\to M,\vec{w}\mapsto\text{proj}_M(\vec{w}) is a linear map.

Proof

We must show that for w⃗1,w⃗2∈Rn\vec{w}_1,\vec{w}_2\in\mathbb{R}^n
projM(c1w⃗1+c2w⃗2)=c1projM(w⃗1)+c2projM(w⃗2)\text{proj}_M(c_1\vec{w}_1+c_2\vec{w}_2)=c_1\text{proj}_M(\vec{w}_1)+c_2\text{proj}_M(\vec{w}_2)
Both w⃗1−projM(w⃗1)\vec{w}_1-\text{proj}_M(\vec{w}_1) and w⃗2−projM(w⃗2)\vec{w}_2-\text{proj}_M(\vec{w}_2) are orthogonal to MM. Therefore, the linear combination of those vectors
c1(w⃗1−projM(w⃗1))+c2(w⃗2−projM(w⃗2))=(c1w⃗1+c2w⃗2)−(c1projM(w⃗1)+c2projM(w⃗2))c_1(\vec{w}_1-\text{proj}_M(\vec{w}_1))+c_2(\vec{w}_2-\text{proj}_M(\vec{w}_2))\\=(c_1\vec{w}_1+c_2\vec{w}_2)-(c_1\text{proj}_M(\vec{w}_1)+c_2\text{proj}_M(\vec{w}_2))
is also orthogonal to MM
Since c1projM(w⃗1)+c2projM(w⃗2)∈Mc_1\text{proj}_M(\vec{w}_1)+c_2\text{proj}_M(\vec{w}_2)\in M, we must have
c1projM(w⃗1)+c2projM(w⃗2)=projM(c1w⃗1+c2w⃗2)c_1\text{proj}_M(\vec{w}_1)+c_2\text{proj}_M(\vec{w}_2)=\text{proj}_M(c_1\vec{w}_1+c_2\vec{w}_2)

The orthogonal complement of a subspace MM of Rn\mathbb{R}^n is
M⊥={w⃗∈Rn ∣ w⃗ is orthogonal to M}M^{\perp}=\{\vec{w}\in\mathbb{R}^n\space|\space\vec{w}\text{ is orthogonal to } M\}
(read "MM perp")

Example 12.4

Find the orthogonal compoenent of the plane in R3\mathbb{R}^3
P={(xyz) ∣ 3x+2y−z=0}P=\{\begin{pmatrix}x\\y\\z\end{pmatrix}\space|\space 3x+2y-z=0\}

First, find a basis for PP
B=⟨(103),(012)⟩B=\langle\begin{pmatrix}1\\0\\3\end{pmatrix},\begin{pmatrix}0\\1\\2\end{pmatrix}\rangle

Steps We have z=3x+2yz=3x+2y, so P={(103)x+(012)y ∣ x,y∈R}P=\left\{\begin{pmatrix}1\\0\\3\end{pmatrix}x+\begin{pmatrix}0\\1\\2\end{pmatrix}y\space|\space x,y\in\mathbb{R}\right\}

A v⃗\vec{v} that is orthogonal to every vector in BB is orthogonal to every vector in span(B)=P\text{span}(B)=P
So this gives two conditions:
(103)⋅(v1v2v3)=0(012)⋅(v1v2v3)=0\begin{array}{cc}\begin{pmatrix}1\\0\\3\end{pmatrix}\cdot\begin{pmatrix}v_1\\v_2\\v_3\end{pmatrix}=0&\begin{pmatrix}0\\1\\2\end{pmatrix}\cdot\begin{pmatrix}v_1\\v_2\\v_3\end{pmatrix}=0\end{array}
This gives a linear system
P⊥={(v1v2v3) ∣ (103012)(v1v2v3)=(00)}P^{\perp}=\{\begin{pmatrix}v_1\\v_2\\v_3\end{pmatrix}\space|\space\begin{pmatrix}1&0&3\\0&1&2\end{pmatrix}\begin{pmatrix}v_1\\v_2\\v_3\end{pmatrix}=\begin{pmatrix}0\\0\end{pmatrix}\}
we therefore must find the nullspace of the matrix
P⊥={(−3−21)t ∣ t∈R}P^\perp=\{\begin{pmatrix}-3\\-2\\1\end{pmatrix}t\space|\space t\in\mathbb{R}\}

For a subspace MM and the orthogonal complement M⊥M^\perp,

  1. M⊥M^\perp is itself a subspace
  2. M∩M⊥={0⃗}M\cap M^\perp=\{\vec{0}\}
  3. For every w⃗∈Rn\vec{w}\in\mathbb{R}^n, w⃗−projM(w⃗)∈M⊥\vec{w}-\text{proj}_M(\vec{w})\in M^\perp
  4. The span of M⊥∪MM^\perp\cup M is all of Rn\mathbb{R}^n
  5. If dimension(M)=k\text{dimension}(M)=k, then dimension(M⊥)=n−k\text{dimension}(M^\perp)=n-k
Proofs
  1. 0⃗∈M\vec{0}\in M and 0⃗⋅v⃗=0\vec{0}\cdot\vec{v}=0 for all v⃗∈M\vec{v}\in M so 0⃗∈M⊥\vec{0}\in M^\perp as well.
    From b) from before, M⊥M^\perp is closed under vector addition and scalar multiplication. Thus, M⊥M^\perp is a subspace of Rn\mathbb{R}^n
  1. From part a), the only vector in MM that is orthogonal to MM is 0⃗\vec{0}, so M∩M⊥={0⃗}M\cap M^\perp=\{\vec{0}\}
  1. By definition of projM(w⃗)\text{proj}_M(\vec{w}), w⃗−projM(w⃗)\vec{w}-\text{proj}_M(\vec{w}) is orthogonal to MM, so w⃗−projM(w⃗)∈M⊥\vec{w}-\text{proj}_M(\vec{w})\in M^\perp
  1. For any w⃗∈Rn\vec{w}\in\mathbb{R}^n, we have w⃗=(w⃗−projM(w⃗))+projM(w⃗)\vec{w}=(\vec{w}-\text{proj}_M(\vec{w}))+\text{proj}_M(\vec{w})
    but w⃗−projM(w⃗)∈M⊥\vec{w}-\text{proj}_M(\vec{w})\in M^\perp and projM(w⃗)∈M\text{proj}_M(\vec{w})\in M
  1. First, suppose dimension(M⊥)=l\text{dimension}(M^\perp)=l
    Then choose orthonormla bases BM=⟨b⃗1,...,b⃗k⟩B_M=\langle\vec{b}_1,...,\vec{b}_k\rangle of MM and BM⊥=⟨b⃗k+1,...,b⃗k+l⟩B_{M^\perp}=\langle\vec{b}_{k+1},...,\vec{b}_{k+l}\rangle of M⊥M^\perp
    BMB_M spans MM and BM⊥B_{M^\perp} spans M⊥⟹⟨b⃗1,...,b⃗k,b⃗k+1,...,b⃗k+l⟩M^\perp\implies\langle\vec{b}_1,...,\vec{b}_k,\vec{b}_{k+1},...,\vec{b}_{k+l}\rangle spans Rn\mathbb{R}^n by (3)
    We consider b⃗i⋅b⃗j\vec{b}_i\cdot\vec{b}_j for i<ji<j:
    If j≤kj\le k, since ⟨b⃗1,...,b⃗k⟩\langle\vec{b}_1,...,\vec{b}_k\rangle is orthonormal, b⃗i⋅b⃗j=0\vec{b}_i\cdot\vec{b}_j=0.
    If k+1≤ik+1\le i, since ⟨b⃗k+1,...,b⃗k+l⟩\langle\vec{b}_{k+1},...,\vec{b}_{k+l}\rangle is orthonormal, b⃗i⋅b⃗j=0\vec{b}_i\cdot\vec{b}_j=0.
    If i≤ki\le k and k+1≤jk+1\le j, then b⃗i∈M\vec{b}_i\in M and b⃗j∈M⊥\vec{b}_j\in M^\perp, so they are perpendicular, thus b⃗i⋅b⃗j=0\vec{b}_i\cdot\vec{b}_j=0.
    So, the family {b⃗1,...,b⃗k,b⃗k+1,...,b⃗k+l}\{\vec{b}_1,...,\vec{b}_k,\vec{b}_{k+1},...,\vec{b}_{k+l}\} is linearly independent, nonzero, and span Rn\mathbb{R}^n. Therefore, k+l=n⟹l=n−kk+l=n\implies l=n-k, finishing the proof.

If MM is a subspace of Rn\mathbb{R}^n, then MM is the orthogonal complement of M⊥M^\perp, i.e. (M⊥)⊥=M(M^\perp)^\perp=M
For every w⃗∈Rn\vec{w}\in\mathbb{R}^n,
w⃗=projM(w⃗)+projM⊥(w⃗)\vec{w}=\text{proj}_M(\vec{w})+\text{proj}_{M^\perp}(\vec{w})

Proof

From the definition of M⊥M^\perp, if v⃗∈M\vec{v}\in M then v⃗\vec{v} is orthogonal to every vector in M⊥M^\perp, so v⃗∈(M⊥)⊥\vec{v}\in(M^\perp)^\perp, and M⊆(M⊥)⊥M\subseteq(M^\perp)^\perp.
Furthermore, we know that dimension(M)+dimension(M⊥)=n\text{dimension}(M)+\text{dimension}(M^\perp)=n and dimension(M⊥)+dimension((M⊥)⊥)=n\text{dimension}(M^\perp)+\text{dimension}((M^\perp)^\perp)=n, so dimension(M)=dimension((M⊥)⊥)\text{dimension}(M)=\text{dimension}((M^\perp)^\perp)
With those two facts, we can conclude that M=(M⊥)⊥M=(M^\perp)^\perp
For the second part, define w⃗⊥=w⃗−projM(w⃗)\vec{w}^\perp=\vec{w}-\text{proj}_M(\vec{w}). Since w⃗−w⃗⊥=projM(w⃗)∈M=(M⊥)⊥\vec{w}-\vec{w}^\perp=\text{proj}_M(\vec{w})\in M=(M^\perp)^\perp, we have that w⃗−w⃗⊥\vec{w}-\vec{w}^\perp is orthogonal to M⊥M^\perp, with w⃗∈M⊥\vec{w}\in M^\perp, so w⃗−w⃗⊥=w⃗−projM⊥(w⃗)⟹w⃗⊥=projM⊥(w⃗)\vec{w}-\vec{w}^\perp=\vec{w}-\text{proj}_{M^\perp}(\vec{w})\implies\vec{w}^\perp=\text{proj}_{M^\perp}(\vec{w})
Finally, w⃗=w⃗⊥+projM(w⃗)=projM⊥(w⃗)+projM(w⃗)\vec{w}=\vec{w}^\perp+\text{proj}_M(\vec{w})=\text{proj}_{M^\perp}(\vec{w})+\text{proj}_M(\vec{w})

Given a subspace M⊆RnM\subseteq\mathbb{R}^n, how can we compute projM(w⃗)\text{proj}_M(\vec{w}) of a vector w⃗∈Rn\vec{w}\in\mathbb{R}^n?
We will suppose the basis for MM is B=⟨b⃗1,...,b⃗k⟩B=\langle\vec{b}_1,...,\vec{b}_k\rangle
If BB is an orthonormal basis, then we know
projM(w⃗)=(w⃗⋅b⃗1)b⃗1+⋯+(w⃗⋅b⃗k)b⃗k=UUTw⃗\text{proj}_M(\vec{w})=(\vec{w}\cdot\vec{b}_1)\vec{b}_1+\cdots+(\vec{w}\cdot\vec{b}_k)\vec{b}_k=UU^T\vec{w}
or equivalently
RepBM(projM(w⃗))=(w⃗⋅b⃗1⋮w⃗⋅b⃗k)\text{Rep}_{B_M}(\text{proj}_M(\vec{w}))=\begin{pmatrix}\vec{w}\cdot\vec{b}_1\\\vdots\\\vec{w}\cdot\vec{b}_k\end{pmatrix}

If BB is an orthogonal basis, then
⟨b⃗1∣b⃗1∣,...,b⃗k∣b⃗k∣⟩\langle\frac{\vec{b}_1}{|\vec{b}_1|},...,\frac{\vec{b}_k}{|\vec{b}_k|}\rangle
is orthonormal.

If BB isn't orthogonal, you could use Gram-Schmidt, but we use a more convenient formula:

Let M⊆RnM\subseteq\mathbb{R}^n be a subspace with basis ⟨b⃗1,...,b⃗k⟩\langle\vec{b}_1,...,\vec{b}_k\rangle and let AA be the matrix whose columns are the b⃗i\vec{b}_i's. Then
projM(v⃗)=c1b⃗1+⋯+ckb⃗k\text{proj}_M(\vec{v})=c_1\vec{b}_1+\cdots+c_k\vec{b}_k
where the cic_i's are the entries of the vector
(ATA)−1AT⋅v⃗(A^TA)^{-1}A^T\cdot\vec{v}
or equivalently,
projM(v⃗)=A(ATA)−1AT⋅v⃗\text{proj}_M(\vec{v})=A(A^TA)^{-1}A^T\cdot\vec{v}

Proof

Given: ⟨b⃗1,...,b⃗k⟩\langle\vec{b}_1,...,\vec{b}_k\rangle is the basis of M⊆RnM\subseteq\mathbb{R}^n and AA is an n×kn\times k matrix with column ii being b⃗i\vec{b}_i
projM(v⃗)∈M⟹projM(v⃗)=c1b⃗1+⋯+ckb⃗k=Ac⃗\text{proj}_M(\vec{v})\in M\implies\text{proj}_M(\vec{v})=c_1\vec{b}_1+\cdots+c_k\vec{b}_k=A\vec{c} where c⃗=(c1⋮ck)\vec{c}=\begin{pmatrix}c_1\\\vdots\\c_k\end{pmatrix}
v⃗−projM(v⃗)\vec{v}-\text{proj}_M(\vec{v}) is orthogonal to every b⃗i\vec{b}_i, which is every row of AT⟹AT(v⃗−projM(v⃗))=0A^T\implies A^T(\vec{v}-\text{proj}_M(\vec{v}))=0
⟹AT(v⃗−Ac⃗)=ATv⃗−ATAc⃗=0⟹c⃗=(ATA)−1ATv⃗\implies A^T(\vec{v}-A\vec{c})=A^T\vec{v}-A^TA\vec{c}=0\implies\vec{c}=(A^TA)^{-1}A^T\vec{v} (check ATAA^TA is invertible)
Thus, projM(v⃗)=Ac⃗=A(ATA)−1ATv⃗\text{proj}_M(\vec{v})=A\vec{c}=A(A^TA)^{-1}A^T\vec{v}

Note that (ATA)−1≠A−1(AT)−1(A^TA)^{-1}\ne A^{-1}(A^T)^{-1} because AA is not square

Example 12.5

Project v⃗=(1−11)\vec{v}=\begin{pmatrix}1\\-1\\1\end{pmatrix} onto the plane P={(xyz) ∣ x+z=0}P=\{\begin{pmatrix}x\\y\\z\end{pmatrix}\space|\space x+z=0\}

A basis for PP is ⟨(010),(10−1)⟩\langle\begin{pmatrix}0\\1\\0\end{pmatrix},\begin{pmatrix}1\\0\\-1\end{pmatrix}\rangle so
A=(01100−1) AT=(01010−1)A=\begin{pmatrix}0&1\\1&0\\0&-1\end{pmatrix}\space A^T=\begin{pmatrix}0&1&0\\1&0&-1\end{pmatrix}
Now, we simply compute A(ATA)−1ATv⃗A(A^TA)^{-1}A^T\vec{v}
ATA=(1002)A^TA=\begin{pmatrix}1&0\\0&2\end{pmatrix}
(ATA)−1=(1001/2)(A^TA)^{-1}=\begin{pmatrix}1&0\\0&1/2\end{pmatrix}
(ATA)−1AT=(0101/20−1/2)(A^TA)^{-1}A^T=\begin{pmatrix}0&1&0\\1/2&0&-1/2\end{pmatrix}
A(ATA)−1AT=(1/20−1/2010−1/201/2)A(A^TA)^{-1}A^T=\begin{pmatrix}1/2&0&-1/2\\0&1&0\\-1/2&0&1/2\end{pmatrix}
Finally
projP(v⃗)=(1/20−1/2010−1/201/2)(1−11)=(0−10)\text{proj}_P(\vec{v})=\begin{pmatrix}1/2&0&-1/2\\0&1&0\\-1/2&0&1/2\end{pmatrix}\begin{pmatrix}1\\-1\\1\end{pmatrix}=\begin{pmatrix}0\\-1\\0\end{pmatrix}

Given a subspace M⊆RnM\subseteq\mathbb{R}^n, the distance from w⃗∈Rn\vec{w}\in\mathbb{R}^n to MM is the smallest possible distance from w⃗\vec{w} to a point on MM
The distance from w⃗\vec{w} to MM is ∣w⃗−projM(w⃗)∣|\vec{w}-\text{proj}_M(\vec{w})|, or equivlaently, ∣w⃗−v⃗∣≥∣w⃗−projM(w⃗)∣|\vec{w}-\vec{v}|\ge|\vec{w}-\text{proj}_M(\vec{w})| for all v⃗∈M\vec{v}\in M

Proof

We know w⃗=projM(w⃗)+projM⊥(v⃗)\vec{w}=\text{proj}_M(\vec{w})+\text{proj}_{M^\perp}(\vec{v})
⟹w⃗−v⃗=(projM(w⃗)−v⃗)+projM⊥(v⃗)\implies\vec{w}-\vec{v}=(\text{proj}_M(\vec{w})-\vec{v})+\text{proj}_{M^\perp}(\vec{v})
Since v⃗∈M\vec{v}\in M, projM(w⃗)−v⃗∈M\text{proj}_M(\vec{w})-\vec{v}\in M, so is orthogonal to w⃗−projM(w⃗)=projM⊥(w⃗)∈M⊥\vec{w}-\text{proj}_M(\vec{w})=\text{proj}_{M^\perp}(\vec{w})\in M^\perp. Therefore, by Pythagorean Theorem,
∣w⃗−v⃗∣2=∣projM(w⃗)−v⃗∣2+∣w⃗−projM(w⃗)∣2|\vec{w}-\vec{v}|^2=|\text{proj}_M(\vec{w})-\vec{v}|^2+|\vec{w}-\text{proj}_M(\vec{w})|^2
So ∣w⃗−v⃗∣≥∣w⃗−projM(w⃗)∣|\vec{w}-\vec{v}|\ge|\vec{w}-\text{proj}_M(\vec{w})|, with equality when projM(w⃗)=v⃗\text{proj}_M(\vec{w})=\vec{v}